Maximizing Multimodal: Exploring the search frontier of text-image models to improve visual find-ability for creatives

“Objective: Describe where and how we have improved the search experience in our product with open source multi-modal models and libraries. Real world examples from the things we have shipped (and decided not to ship) to production, including AB test results of our relevancy changes.

Outline:

1. Cover the architecture of our open source hybrid search stack at Eezy (Elasticsearch, FAISS, PyTorch models)
2. Describe the capabilities and limitations of openCLIP (and vector embeddings at large) for retrieval tasks and current pain points and work arounds we have engineered.
3. Highlight meaningful stops on our product roadmap from the last 2 years of deploying features into production.
4. Describe notable missteps and surprises uncovered along the way, so people see it’s not all roses in the AI powered future.
5. Demo BORGES, a novel search framework that allows users to search with multiple queries in multiple modalities for a nuanced navigation of the catalog to find exactly what they need

Audience:

• Anyone curious about real-world results we have extracted from AI
• Search practitioners developing hybrid search applications
• PyTorch and transformers enthusiasts interested in applications in vector space”

Nathan Day

Eezy